Hello everyone,

The company I work for has WCF Web Services hosted on Windows Azure in a medium sized webrole with two instances. We call the services from a WPF Application. 99.9% of the time the calls from the client application to the WCF Web Services work correctly, but every once in a while trying to call a web service method returns the following exception:

Source
mscorlib

Message
There was no endpoint listening at https://website/ServiceName.svc that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details.

StackTrace

Server stack trace:
at System.ServiceModel.Channels.HttpOutput.WebRequestHttpOutput.GetOutputStream()
at System.ServiceModel.Channels.HttpOutput.Send(TimeSpan timeout)
at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.SendRequest(Message message, TimeSpan timeout)
at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at ApplicationName.ServiceName.IServiceName.WebServiceMethodName(Guid sessionToken)
at ApplicationName.WindowName.ClientApplicationMethodName(Object sender, DoWorkEventArgs e)
at System.ComponentModel.BackgroundWorker.OnDoWork(DoWorkEventArgs e)
at System.ComponentModel.BackgroundWorker.WorkerThreadStart(Object argument)

TargetSite
Void HandleReturnMessage(System.Runtime.Remoting.Messaging.IMessage, System.Runtime.Remoting.Messaging.IMessage)

Source
System

Message
The remote name could not be resolved: 'website' (website where web services are hosted, same as "website" from There was no endpoint listening at https://website/ServiceName.svc above)

StackTrace
at System.Net.HttpWebRequest.GetRequestStream(TransportContext& context)
at System.Net.HttpWebRequest.GetRequestStream()
at System.ServiceModel.Channels.HttpOutput.WebRequestHttpOutput.GetOutputStream()

TargetSite
System.IO.Stream GetRequestStream(System.Net.TransportContext ByRef)

Although the exceptions are very rare, I've personally replicated this when running the client application on my 64-bit Windows 7 machine and its Windows XP Virtual Machine. Our customers have seen this occur when running the client application on Windows Vista & Windows XP machines.

The exceptions have occured in across multiple web services and methods, for both normal and async calls.

I find these exceptions confusing because normally a "no endpoint listening" exception occurs when trying to get the configuration right, but in this case the configuration is correct and the web services usually work great. Because the exceptions are so rare it's difficult to replicate. Also, since we're hosting the services on a medium sized Azure webrole with two instances (and we're not all that busy), I don't think it's a case of the web services being overloaded.

I checked the Azure IIS logs for the date and time when I replicated this exception but couldn't find anything of interest. (if the client is saying it couldn't resolve the name where the web services are hosted then I'm guessing the request didn't make it through to Azure and thus didn't appear in the IIS logs...)

I originally posted about this in the WCF forms but they sent me over here. Is there anything in either the WCF or Azure configurations I should check? Is there any more information I can provide you guys?

Thanks for the help,

ZenkeiRich

• Changed type ZenkeiRich Wednesday, August 29, 2012 8:14 AM
• Edited by ZenkeiRich Monday, September 03, 2012 4:06 AM

How may users will access this service, do you think 2 medium instances is enough for your customers?

Two medium stances should be more than enough for our current number of users. For the webrole in question the CPU max is 11.5% with an average of 2.6%, Network In has a max of 35megs with an average of 5.7k, Network Out max is 11.6megs, average is 2k. IIRC you should upgrade a role when the CPU% gets around 70%, and medium instances have an allocated bandwith of 200megs.

Hi ZenkeiRich,

Just to be sure here... Can you confirm that your're closing the WCF ServiceClients?

Carlos,

We use "using" for all of our WCF ServiceClients. To the best of my understanding this should close and dispose the ServiceClients.

Example:

using (ServiceName.ServiceNameClient client = new ServiceName.ServiceNameClient())

{

//call web services

}

Hey everyone,

In addition to the "no endpoint listening" exception, over the weekend our WCF web services have returned different exceptions to the client. Judging from the inner exceptions the first is "An existing connection was forcibly closed by the remote host". The original exception mentions a possible mismatch of the security binding between client and server, but I beleive if this was the case the web service calls would fail 100% of the time.

Source
mscorlib

Message
An error occured while making the HTTP request to https://website/ServiceName.svc . This could be due to the fact that the server certificate is not configured properly with HTTP.SYS in the HTTPS case. This could also be caused by a mismatch of the security binding between the client and the server.

StackTrace
 System.ServiceModel.Channels.HttpChannelUtilities.ProcessGetResponseWebException(WebException webException, HttpWebRequest request, HttpAbortReason abortReason)
 System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)
 System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
 System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout)
 System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
 System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
 System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]:
 System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
 System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
 ClientApplicationName.ServiceName.IServiceName.WebServiceMethodName(Project project, Guid sessionToken)
 ClientApplicationName.Utils.ProjectUtils.SaveProject(Project project)
 ClientApplicationName.MainWindow.clientBackgroundWorker_DoWork(Object sender, DoWorkEventArgs e)
 System.ComponentModel.BackgroundWorker.OnDoWork(DoWorkEventArgs e)
 System.ComponentModel.BackgroundWorker.WorkerThreadStart(Object argument)

TargetSite
Void HandleReturnMessage(System.Runtime.Remoting.Messaging.IMessage, System.Runtime.Remoting.Messaging.IMessage)

Source
System

Message
The underlying connection was closed: An unexpected error occurred on a send.

StackTrace
 System.Net.HttpWebRequest.GetResponse()
 System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)

TargetSite
System.Net.WebResponse GetResponse()

Source
System

Message
Unable to write to the transport connection. An existing connection was forcibly closed by the remote host

StackTrace
 System.Net.Sockets.NetworkStream.MultipleWrite(BufferOffsetSize[] buffers)
 System.Net.Security._SslStream.StartWriting(SplitWritesState splitWrite, SplitWriteAsyncProtocolRequest asyncRequest)
 System.Net.Security._SslStream.ProcessWrite(BufferOffsetSize[] buffers, SplitWriteAsyncProtocolRequest asyncRequest)
 System.Net.TlsStream.MultipleWrite(BufferOffsetSize[] buffers)
 System.Net.Connection.Write(ScatterGatherBuffers writeBuffer)
 System.Net.ConnectStream.ResubmitWrite(ConnectStream oldStream, Boolean suppressWrite)

TargetSite
Void MultipleWrite(System.Net.BufferOffsetSize[])

Source
System

Message

An existing connection was forcibly closed by the remote host

StackTrace
 System.Net.Sockets.Socket.MultipleSend(BufferOffsetSize[] buffers, SocketFlags socketFlags)
 System.Net.Sockets.NetworkStream.MultipleWrite(BufferOffsetSize[] buffers)

TargetSite
Void MultipleSend(System.Net.BufferOffsetSize[], System.Net.Sockets.SocketFlags)

Hey everyone,

Another exception returned was a "operation timed out". The web service method called when this exception occured is a fast method that takes seconds to complete, so it's weird that it would hit our 20 minute timeout...

Source
mscorlib

Message
The request channel timed out while waiting for a reply after 00:20:00. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout.

StackTrace

Server stack trace:
 System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
 System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout)
 System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
 System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
 System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]:
 System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
 System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
 WebServiceMethodName(Guid sessionToken)
 ApplicationName.MainWindow.clientBackgroundWorker_DoWork(Object sender, DoWorkEventArgs e)
 System.ComponentModel.BackgroundWorker.OnDoWork(DoWorkEventArgs e)
 System.ComponentModel.BackgroundWorker.WorkerThreadStart(Object argument)

TargetSite
Void HandleReturnMessage(System.Runtime.Remoting.Messaging.IMessage, System.Runtime.Remoting.Messaging.IMessage)

Source
System.ServiceModel

Message
'https://website/ServiceName.svc' The request operation did not complete within the allotted timeout of 00:19:59.3870000. The time allotted to this operation may have been a portion of a longer timeout.

StackTrace
 System.ServiceModel.Channels.HttpOutput.WebRequestHttpOutput.GetOutputStream()
 System.ServiceModel.Channels.HttpOutput.Send(TimeSpan timeout)
 System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.SendRequest(Message message, TimeSpan timeout)
 System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)

TargetSite
System.IO.Stream GetOutputStream()

Source
System

Message
The operation has timed out.

StackTrace
 System.Net.HttpWebRequest.GetRequestStream(TransportContext& context)
 System.Net.HttpWebRequest.GetRequestStream()
 System.ServiceModel.Channels.HttpOutput.WebRequestHttpOutput.GetOutputStream()

TargetSite
System.IO.Stream GetRequestStream(System.Net.TransportContext ByRef)

Did you ever solve this issue?  We just started deploying our winforms application and I noticed the same problem.  We cache WCF connections for 10 seconds.  If our client application makes a call, and the connection is over 10 seconds old, it would be closed and a new one opened.  After long periods of inactivity, I get a "There was no endpoint listening at" application exception.  It is very strange that I cannot trap this message.

• Proposed as answer by M Landy Tuesday, July 02, 2013 9:25 PM

Was this ever solved? I'm getting the same issues on IIS 8.  Not sure how to troubleshoot as there is no consistency and no errors on IIS.

Hi,

We are experiencing the same issues as well. 

Once in a while we get these exception 'bursts'. These exception sequences are always thrown from one client machine at a time. Looks like a temporary connectivity issue on the client side.

Did you check if it happens on a single client machine each time?

The timeouts + no endpoint + binding issues all come down to temporary connectivity problems. 

Since we have many clients running all the time simultanously, but these exception sequences only come from one client at a time, I'm almost sure its connectivity loss on the client side.

Hi,

We are having the same error using HTTPS and WCF on IIS 8. A8 Machine IaaS.

Tried a lot of things and nothing... still happen.

Yesterday I read some problem with request limits and others configuration of serviceThrottling of WCF but nothing... the error still happens. 

Best regards,

• Edited by Bruno Kenj CL 12 hours 7 minutes ago